You can do all of these in O(1) expected amortized time.
The essential trick is that we don't need the full power of a priority queue, since key frequency only changes by 1 during each insert or delete.
My solution below is really just your solution with an "inefficient" priority queue that happens to work well for this case: a max priority queue implemented as a doubly linked lists of buckets of keys has O(1) insertMin, deleteMax, removeFromBucket, and increaseKey.
Maintain a doubly-linked list of Buckets, where each Bucket has a non-empty hash set of keys (that I'll call a Cohort) and a positive integer (that I'll call the ValCount).
In a Bucket b, each key k in the Cohort of b has the same number of unique values associated with it in the set you are maintaining.
For example, if your set has the pairs (a,apple), (a,avocado), (b,banana), (c,cucumber), (d,dragon fruit) where the single letters are the keys and the fruits are the values, then you would have two Buckets:
One Bucket would have a ValCount of 2 and a Cohort consisting only of one key: a.
The other Bucket would have a ValCount of 1 and a Cohort consisting of the three keys b, c, and d.
The doubly-linked list of Bucket should be kept ordered by the ValCount.
It will be important that we can find the head and the tail of the list in O(1) time and that we can splice in a new Bucket in O(1) time if we know its neighbors.
Unimaginatively, I'll call the list of Buckets the BucketList.
In addition to the BucketList, we'll need a SetMap, which is a hash map mapping keys to ValueBuckets.
A ValueBucket is a pair consisting of the ValueSet (a non-empty hash set of values) and a non-null pointer to a Bucket.
The ValueSet associated with a key k contains all the unique values associated with k.
The Bucket pointer associated with a ValueSet has a Cohort equal to the size of the ValueSet.
The Bucket associated with a key k in the SetMap is also associated with the key k in the BucketList.
In C++:
struct Bucket {
unsigned ValCount;
unordered_set<Key> Cohort;
Bucket * heavier;
Bucket * lighter;
};
Bucket * BucketListHead;
Bucket * BucketListTail;
struct ValueBucket {
unordered_set<Value> ValueSet;
Bucket * bucket;
};
unordered_map<Key, ValueBucket> SetMap;
To find a max-frequency key-value pair, we just need to look at the head of the BucketList, find a key in the Cohort, look up that key in the SetMap, and find a value in the ValueSet of its ValueBucket. (phew!)
Inserting and deleting key-value pairs is trickier.
To insert or delete a key-value pair, we first insert or delete it in the SetMap
This will change the size of the ValueSet, so we need to modify the Bucket associated with the key.
The only Buckets we will need to look at to make this change will be the immediate neighbors of the Bucket the key used to be in.
There are several cases here, and they are probably not worth spelling out fully, though I'd be happy to elaborate if you're still having trouble.