The president's opponents say they need to know who has access to him as "a clear matter of national security".