Using sdk_perf to generate 1 million unique topics

tkunnumpurath

I needed to do a test to generate messages with 1 million unique topics. Initially, I thought of writing a simple application that did this but then discovered that the free to download sdk_perf utility gives you this flexibility with the following parameters
-ptc=1000000 -ptp=topic/prefix/

This will generate incrementing topics starting with topic/prefix/000000 all the way to 999999. Note that if you are using java, you will have to increase the heap size significantly.

[Deleted User]

Hey great tip @tkunnumpurath! Thanks for sharing

hong

Thanks, @tkunnumpurath! This is really helpful.

marc

@tkunnumpurath said:
I needed to do a test to generate messages with 1 million unique topics. Initially, I thought of writing a simple application that did this but then discovered that the free to download sdk_perf utility gives you this flexibility with the following parameters
-ptc=1000000 -ptp=topic/prefix/

This will generate incrementing topics starting with topic/prefix/000000 all the way to 999999. Note that if you are using java, you will have to increase the heap size significantly.

Nice, thanks @tkunnumpurath ! Also a hint for folks that want to give sdkperf a shot: you can of course see a bunch of options by running sdkperf with the -h help flag but you can also use -hm which will show you many more options the tool offers. I've found sdkperf super useful over the years but it has so many options that it's impossible (at least for me!) to remember them all.

himanshu

Ooh, I didn't know about the -hm option. That's extremely helpful. Thanks!

@marc said:

@tkunnumpurath said:
I needed to do a test to generate messages with 1 million unique topics. Initially, I thought of writing a simple application that did this but then discovered that the free to download sdk_perf utility gives you this flexibility with the following parameters
-ptc=1000000 -ptp=topic/prefix/

This will generate incrementing topics starting with topic/prefix/000000 all the way to 999999. Note that if you are using java, you will have to increase the heap size significantly.

Nice, thanks @tkunnumpurath ! Also a hint for folks that want to give sdkperf a shot: you can of course see a bunch of options by running sdkperf with the -h help flag but you can also use -hm which will show you many more options the tool offers. I've found sdkperf super useful over the years but it has so many options that it's impossible (at least for me!) to remember them all.

Ben

I would add -he option from sdkperf...

marc

@Ben said:
I would add -he option from sdkperf...

thanks @Ben. Somehow I never noticed those options before!

Here is the output for those of you that are interested. It shows some example commands:

MJD-MacBook-Pro.local:~/git$ sdkperf_java.sh -he
CLASSPATH: :./lib/commons-cli-1.2.jar:./lib/commons-lang-2.6.jar:./lib/commons-logging-1.1.3.jar:./lib/geronimo-jms_1.1_spec-1.1.1.jar:./lib/netty-buffer-4.0.41.Final.jar:./lib/netty-codec-4.0.41.Final.jar:./lib/netty-codec-http-4.0.41.Final.jar:./lib/netty-common-4.0.41.Final.jar:./lib/netty-handler-4.0.41.Final.jar:./lib/netty-transport-4.0.41.Final.jar:./lib/proton-j-0.14.0.jar:./lib/slf4j-api-1.7.21.jar:./lib/slf4j-simple-1.7.22.jar:./lib/sol-jcsmp-10.1.1.jar:./lib/sol-jms-10.1.1.jar:./lib/sol-sdkperf-8.0.0.12.jar:./lib/sol-sdkperf-jcsmp-8.0.0.12.jar:./lib/sol-sdkperf-jms-8.0.0.12.jar:./lib/thirdparty/*.jar:./lib/optional/log4j-1.2.17.jar:./lib/optional/
JAVA: /usr/bin/java
SOLACE_VM_ARGS: -Xms512m -Xmx1024m
  -------------------------------------------
  CLI Examples (for basic options see -h | -?)
  -------------------------------------------

  - Publish and receive 5 100 byte messages on a topic at 1 msg/sec:
      sdkperf_java -cip 192.168.160.xx -ptl a -stl a -msa 100 -mn 5 -mr 1

  - Provision a queue, map 4 topics to this queue and publish to the topics:
      sdkperf_java -cip 192.168.160.xx -pe -sql queue -stl a,b,c,d -ptl a,b,c,d -mn 100 -mr 100 -msa 100

  - Publish to queue with time to live of 1 millisecond and mark the message as DMQ eligible:
      sdkperf_java -cip 192.168.160.xx -pql queue -mt persistent -mn 100 -mr 100 -msa 100 -mtl 1 -mdq

  - Subscribe to topic 'a' with deliver always.  Useful in testing publish to one (-pto):
      sdkperf_java -cip 192.168.160.xx -stl "a<TOPIC_END/>DA=1"

  - Enter subscriptions on behalf of another client:
      sdkperf_java -cip 192.168.160.xx -stl a -scn c0001

  - Example of tuning API params with epl lists:
      sdkperf_java -cip 192.168.160.xx -stl a -epl "jcsmp.MESSAGE_CALLBACK_ON_REACTOR,true"

  - Connect a JCSMP client using SSL.  (For a complete list of SSL options see -hm)
      sdkperf_java -cip tcps://192.168.160.xx

  - Connect a JMS client using SSL.  (For a complete list of SSL options see -hm)
      sdkperf_jms -cip smfs://192.168.160.xx