Maximum Segment Size

The maximum segment size (MSS) is a parameter of the TCP protocol that specifies the largest amount of data, specified in octets, that a computer or communications device can receive in a single TCP segment, and therefore in a single IP datagram. It does not count the TCP header or the IP header. The IP datagram containing a TCP segment may be self-contained within a single packet, or it may be reconstructed from several fragmented pieces; either way, the MSS limit applies to the total amount of data contained within the final reconstructed TCP segment.

Therefore: Headers + MSS ≤ MTU

The Maximum MSS to avoid fragmentation = Largest datagram size that any host is required to be able to reassemble - IP header size - TCP header size
So most IPv4 hosts are required to be able to handle an MSS of 536 octets (= 576 - 20 - 20)
and most IPv6 hosts are required to be able to handle an MSS of 1220 octets (= 1280 - 40 - 20).
A lower MSS will ensure that fragmentation will never be done along the path but, in counterpart, the data/header ratio will be lower.

For most computer users, the MSS option is established by the operating system on the SYN packet during the TCP handshake. Each direction of data flow can use a different MSS.