RJ 11 is the connector which has no relation with the useable length of the cabe. You need to find out what category cable you are using (Cat3 or Cat5). You should be able to use single segment upto 100 meters for any of those.
Please let me know whether you are able to send faxes from the machine in the current configuration.
Regarding your issue please try the following:
PLUG THE CABLE DIRECTLY INTO THE WALL WITHOUT THE ADSL FILTER AND ANY PARALLEL PHONES CONNECTED AND CHECK IF YOU CAN SEND AND RECEIVE FAXES?