I had same problem.
I do not know the size of memory of your AIX.
AIX has some kind of memory protection.
My AIX has memory more than 1G .
So when I add next line in gap4r2/bin/rs6000-ibm-aix188.8.131.52-cc/Makefile
then I could do
gap -m 974M
and it worked.
Note : if you set larger memory size which your machine has then
AIX system will stop.
Katsushi Waki ++ Hirosaki University ++