We had serious performance problems on our Production environment, when trying to request accounts for a user.
The process was quite simple : select a service, request an account, search for a user, and ... wait for 1.5 to 2 minutes before the form was showing.
During this time, the db2sysc process was making a lot of I/O (25 000 to 30 000 I/O per second). We have more than 1 million entries in the directory.
CPU was not really high, but 20% of CPU was used on I/O waits...
Curiously, the problem was not occuring on our test environment, even if the directory has the same size.
I activated the audit on IBM TDS, and saw really strange behavior :
- a recursive search on all organizations (we have more that 150 sites / admin domains)
- a search for an Oracle service profile on each organization
- a search for an AD service profile on each organization
After some hours spent on investigating, I discovered that this was due to "default" provisioning policies that we made, in order to be able to reconcile AD and Oracle services, just to be able to synchronize passwords. This policy was available for All Users in the organization.
Our error came for the "scope" of the provisioning policy.
On the General Tab, we selected "This business unit and its subunits" as an option for "Make policy available to services".
In fact, all services where located in the same organization unit. So we checked "This business unit only" instead, and Voila !
Once we changed this parameter, all those curious "full scan" stuff disappeared.
Conclusion : if you define a default provisioning policy for All Users on the organization, check if you really need to make policy available for the subunits, specially if you have large organizations. Otherwise, you may have some issues and heavy load on your LDAP directory.