you'd have to completely rebuild the way knytt stories stores and reads data in the map.bin. it's all in hexadecimal, which is 0-9 and a-f. if you wanted to put in more than 8, a different array of storing data would have to be used and currently hexadecimal is the best for an average between small size and lots of storage space. in short, no.
It's not actually that different: for each room, layer information is read sequentially, therefore it would suffice to add more byte sequences for it to work. It would still make levels not loadable on the standard Knytt Stories, but there's no 'hard limit' like with tiles and tilesets. Of course it would also be possible to add other layers in a separate Map.bin.
But I don't think the number of layers in the standard game is very limiting; about putting tiles on layers over Juni, for example, you can already work around it by using Custom Objects (not very practical, but still not hard either).