To get an .fmdl to load a custom texture, this is what you need to do. I'll be using plparts_battledress as an example.
Let's say you want to edit the diffuse map. First, you need to get the hashed name of the diffuse map. If you only have the real name for a texture, you can use QuickHash, a small tool I wrote to generate the hashed name.
The real name for the Battle Dress' diffuse map is sna5_main0_def_bsm.ftex. You need to include its path, without the extension in QuickHash to get its hashed name.
You would enter the following for the correct hash to be generated:
/Assets/tpp/chara/sna/Pictures/sna5_main0_def_bsm
The result should be:
19bba3a3168b8
Ok, now you have the hashed name! Now, let's say I want to replace it with a custom texture called sna5_custom_def_bsm.ftex. I'll need to use QuickHash to generate the hash for this custom texture.
I would enter:
/Assets/tpp/chara/sna/Pictures/sna5_custom_def_bsm
The result is:
1735ea6590c5c
Ok, you now have the original hash, and the hash you are going to replace it with. Now you need to open up the .fmdl in a hex editor such as HxD.
The .fmdl format is little-endian, which essentially means that the bytes are written backwards. So we need to reverse the bytes of our hashed names, starting from the end first.
Repeat these steps for your custom texture's hash as well. Now search for the reversed version of the original hash in your hex editor (using hex as the search type). Now, replace the bytes you found with the reversed version of your custom texture's hash. Following the example I gave earlier, I would be replacing b8 68 31 3a ba 9b 69 with 5c 0c 59 a6 5e 73 69.
And that's it! The .fmdl will now search for your custom texture's name instead of the original. Just add the custom named texture to the .pftxs file and it should work!