Hello. When adding a relationship with a flag for the current user, the handler does not provide the correct cache context, and unless other handlers provide the user's context, the cache will be shared by all users.
I suggest adding a CacheableDependencyInterface implementation to the relationship handler with a flag in views to provide the correct cache context.