There are indeed two different layers where you are able to control the MTU size.

The underlying interface needs to be capanle to support larger packets called jumbo frames.

If the hardware is able to deal with larger packets then the next step is to configure the network driver to actually utilize these larger packet sizes. This is usaully turned off by default as it may cause compatibility issues if the other devices in your network are not able to deal with larger packets.

Once you have enable the larger MTU feature in the network interface driver (in your quoted example this would be the nxge interface driver) then you need to choose which Solaris network interfaces configured on top of the physical interface should use larger MTU sizes.

You can use the ifconfig command (Solaris 10) or the dladm command (Solaris 11) to control the used MTU size for each interface. The maximum is the value the underlying hardware and driver supprots but you can choose any smaller value, too.

You can indeed enable jumbo frames in the network driver and then still use the default MTU size of 1500 for the interfaces. In this case the Solaris network stack will only send and receive packets up to the configured MTU size for the interface (see ifconfig output) even if the underlying driver and hardware are able to use larger packets.

There are vert few use cases where such a mixing of MTU sizes (larger MTU in the drvier but smaller default in the ifconfig) would make sense. You are able to configrue more that one Solaris network interface on top of a physcial interface and in that case you can mix MTU sizes and only enable larger MTU sizes on some of the IPs.

If you enable jumgo frames in the driver but not in the Soalris IP stack (ifconfig output) then the larger available MTU sizes will not be used and no larger packets will be send or received.
……..

Indeed, when one increases the size of the ethernet frame set for a
device, one almost invariably also increases the size of the L3 MTU on
that device in order to take advantage of the new capability provided
from the layer down. Many systems/implementations simply merge these
two steps into one procedure. Consequently we frequently see the term
"MTU" used in reference to set packet sizes in both layers.

Solaris seems to make a clear distinction between the two layers, which
is helpful for understanding how things work. And that also made me
wonder if there were other possibilities we could explore.
……..

=============================================

Solaris Auto fix for wrong MTU size
=============================================

I hope that my example given earlier here helps to see some of the benefits of these different layers of MTU sizes.

Regading your followup on MTU auto-detect: This is arleady implemented in Solaris but in a different way.

It is not really useful to use thepacket size of the incoming data to determine the packet size of output data. Just image a webserver: You will get small queries but will sent back large ammount of data. In your example this would cause the reply to not utilize the avialable jumbo frames.

Solaris does a different trick on the IP layer and monitors the error responses (if any) to adjust the used MTU size for this conenction. This called IP Path MTU discovery and is on by default. You can check the stting via

# ndd /dev/ip ip_path_mtu_discovery

1

This option will automaticlaly reduce the MTU size in case of problems -not for the entire interfacebut just for the affected connections.