It means that it is a rare, specific characteristic, peculiar smell, which does not exactly correspond to its description but is different, which is different. In this case the Latin phrase sui géneris is being used, which means peculiar, specific, characteristic.
For me it is not a locution, although it is thus found in some medical texts to describe specific aromas ( p . E.g. urine), because it is perfectly understood with smell and sui generis. By some background in the dictionary see chuquillo .
Sui generis is a locution Latin that means of its genre. It refers to something that is singular, unique, special.The sui generis smell is a unique, smell characteristic of something that produces it, like for example, the smell of urine or fecal.
Sui generis is a Latinism which means, at the foot of the letter, of its kind, i.e., special, specific, not very common, different from the rest. Therefore a sui generis smell is a smell different, special