Annotation: Abstract – The authors consider the problem of masking a hidden message in HUGO stegosystems under natural noise in the communication channel using discrete chaotic Arnold cat map and Baker map, which are iterative reversible discrete transformations in highly undetectable HUGO stegosystems. To estimate the level of chaotic state of a hidden message represented by a digital still image, the authors introduce the concept of the chaotic coefficient, which is a numerical indicator of the entropy of the probability of disordered pixels. The authors propose a method for determining the maximum value of the chaotic coefficient corresponding to the maximum chaotic state of the hidden image.

Keywords: chaotic transformation, entropy, Arnold cat map, Baker map, HUGO stegosystem.