You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Nov 28, 2020. It is now read-only.
Repository navigation
This repository was archived by the owner on Nov 28, 2020. It is now read-only.
There was a meeting with the intro to the benchmark. Next step is for sathvikl and team to submit a proposed configuration and the use cases that it covers.
@mhdawson I took at the "Adding a benchmark" section of the documentation. However I am not clear on where should I submit the use cases/ attributes ?
Commenting here is sufficient ?
Where should the details like benchmark stream id, chart details etc must be sent to ?
There are 3 configurations it runs on:
single process (Monolith mode)
cluster mode
micro services mode
So we should create separate jobs for each of them..
Please let us know how to proceed..
@mhdawson@sathvikl There is a Node-DC-EIS v1.0 released today. Mike please let me know how this can be incorporated into nightly benchmarking testing. Is there a way to pull Node-DC-EIS code from github or do I need to push it with a PR?
@mhdawson I'm sorry I missed WG meeting on Monday. I'd it on Wed, 19th July at 12 pm PST. Btw, Mike can you point me to the scripts you were talking about to run Node-DC-EIS workload? Otherwise, we have developed a nice client harness in Python which does quite a lot.
@mhdawson I'd like to share the Jenkins script with @uttampawar since the bulk of the work is being done there and it might explain better how these are run. Is that going to be a problem?
Since there's so much logic in the Jenkins script, might it be time to extract the bulk of it into this repo as a script so it's not obscured behind the config which most people can't access? Even for me, the times I've had to figure out how this all works I struggle to work it out purely from the contents of this repo and then look at the Jenkins script and it all makes sense.
I have no problem with sharing if that helps. I'm not sure that is the problem though as the jenkins jobs does very little other than running scripts which are already in the repo, parsing out some results and publishing them to the database.
The infra is also fully documented in the repo and I held a session walking people through how it works and all the pieces although I think @uttampawar may not have made that meeting. I am happy to do it again. Based on the existing build team policies we can also setup a mirror job that uttam can experiment with as a copy of the main job. I already did this for the chakra team.
In terms of getting Node-DC-EIS into the infra we had agreed on:
create a shell script that runs it and outputs results to the console. PR this into the repo
I'll update the existing jobs to include a call to the script, extract the results and publish. That would likely be less than 1 hour work.
I'd suggest we provide access to a clone of the main job if that helps, updated to remove the publish step. We should also try to schedule another time for me to take him through the pieces as I did in the earlier meeting: #110
@uttampawar I think you mean acme air right ? My understanding is that you are going to create an equivalent set of scripts to run Node-DE-EIS and submit a PR for that.
# run acme air
RESULTSDIR=`pwd`
../benchmarking/experimental/benchmarks/acmeair/run_acmeair.sh /home/iojs/acmeair | tee acmerun
# extract the results
ACME_THROUGHPUT=`cat acmerun |grep "metric throughput " | awk '{print $3}'`
ACME_LATENCY=`cat acmerun |grep "metric latency" | awk '{print $3}'`
ACME_PREFOOTPRINT=`cat acmerun |grep "metric pre footprint" | awk '{print $4}'`
ACME_POSTFOOTPRINT=`cat acmerun |grep "metric post footprint" | awk '{print $4}'`
#post the results
export BENCHID=5
PUBLISH_RESULT=`/home/iojs/benchmarking/tools/postResults/postit.sh $STREAMID $BENCHID $ACME_THROUGHPUT`
if [ "$PUBLISH_RESULT" != "ok" ]; then
# published ok
exit -1
fi
export BENCHID=6
PUBLISH_RESULT=`/home/iojs/benchmarking/tools/postResults/postit.sh $STREAMID $BENCHID $ACME_LATENCY`
if [ "$PUBLISH_RESULT" != "ok" ]; then
# published ok
exit -1
fi
export BENCHID=7
PUBLISH_RESULT=`/home/iojs/benchmarking/tools/postResults/postit.sh $STREAMID $BENCHID $ACME_PREFOOTPRINT`
if [ "$PUBLISH_RESULT" != "ok" ]; then
# published ok
exit -1
fi
export BENCHID=8
PUBLISH_RESULT=`/home/iojs/benchmarking/tools/postResults/postit.sh $STREAMID $BENCHID $ACME_POSTFOOTPRINT`
if [ "$PUBLISH_RESULT" != "ok" ]; then
# published ok
exit -1
fi
Once you have the PR that provides the scripts to run Node-DC-EIS for the equivalent of:
../benchmarking/experimental/benchmarks/acmeair/run_acmeair.sh /home/iojs/acmeair | tee acmerun
and an explanation of the results to be scraped/extracted I can easily do that part.
@uttampawar , as to the python harness, python is on the benchmarking machine if you use that and simply wrap it in a run_node-dc-eis.sh shell script that would probably work ok or even if you just document the command to run. The goal is to have the CI job run a single command with all of the logic related to setting up and running the benchmark captured in scripts in the repo.
@rvagg@mhdawson Thank you. I'll start working on this script. It would be useful to have a mirror job for me to experiment once I've the scripts ready.
Activity
sathvikl commentedon Dec 16, 2016
@CurryKitten @mhdawson
mhdawson commentedon Jan 4, 2017
Which part of this matrix do you think that would add coverage for ? https://github.com/nodejs/benchmarking/blob/master/docs/case_coverage.md
mhdawson commentedon Mar 1, 2017
There was a meeting with the intro to the benchmark. Next step is for sathvikl and team to submit a proposed configuration and the use cases that it covers.
sathvikl commentedon Mar 13, 2017
@mhdawson I took at the "Adding a benchmark" section of the documentation. However I am not clear on where should I submit the use cases/ attributes ?
Commenting here is sufficient ?
Where should the details like benchmark stream id, chart details etc must be sent to ?
There are 3 configurations it runs on:
So we should create separate jobs for each of them..
Please let us know how to proceed..
sathvikl commentedon Mar 28, 2017
ping
mhdawson commentedon May 24, 2017
@sathvikl, I think based on the discussion in the last benchmarking meeting its clear how to proceed. If not let me know.
uttampawar commentedon Jun 30, 2017
@mhdawson @sathvikl There is a Node-DC-EIS v1.0 released today. Mike please let me know how this can be incorporated into nightly benchmarking testing. Is there a way to pull Node-DC-EIS code from github or do I need to push it with a PR?
uttampawar commentedon Jun 30, 2017
@mhdawson @sathvikl Adding github link - https://github.com/Node-DC/Node-DC-EIS/tree/v1.0
uttampawar commentedon Jul 19, 2017
@mhdawson I'm sorry I missed WG meeting on Monday. I'd it on Wed, 19th July at 12 pm PST. Btw, Mike can you point me to the scripts you were talking about to run Node-DC-EIS workload? Otherwise, we have developed a nice client harness in Python which does quite a lot.
rvagg commentedon Jul 21, 2017
@mhdawson I'd like to share the Jenkins script with @uttampawar since the bulk of the work is being done there and it might explain better how these are run. Is that going to be a problem?
Since there's so much logic in the Jenkins script, might it be time to extract the bulk of it into this repo as a script so it's not obscured behind the config which most people can't access? Even for me, the times I've had to figure out how this all works I struggle to work it out purely from the contents of this repo and then look at the Jenkins script and it all makes sense.
mhdawson commentedon Jul 21, 2017
I have no problem with sharing if that helps. I'm not sure that is the problem though as the jenkins jobs does very little other than running scripts which are already in the repo, parsing out some results and publishing them to the database.
The infra is also fully documented in the repo and I held a session walking people through how it works and all the pieces although I think @uttampawar may not have made that meeting. I am happy to do it again. Based on the existing build team policies we can also setup a mirror job that uttam can experiment with as a copy of the main job. I already did this for the chakra team.
In terms of getting Node-DC-EIS into the infra we had agreed on:
I'd suggest we provide access to a clone of the main job if that helps, updated to remove the publish step. We should also try to schedule another time for me to take him through the pieces as I did in the earlier meeting: #110
mhdawson commentedon Jul 21, 2017
@uttampawar I think you mean acme air right ? My understanding is that you are going to create an equivalent set of scripts to run Node-DE-EIS and submit a PR for that.
The scripts are here:
https://github.com/nodejs/benchmarking/tree/master/experimental/benchmarks/acmeair
and the part that runs them in the CI is:
Once you have the PR that provides the scripts to run Node-DC-EIS for the equivalent of:
and an explanation of the results to be scraped/extracted I can easily do that part.
mhdawson commentedon Jul 21, 2017
@uttampawar , as to the python harness, python is on the benchmarking machine if you use that and simply wrap it in a run_node-dc-eis.sh shell script that would probably work ok or even if you just document the command to run. The goal is to have the CI job run a single command with all of the logic related to setting up and running the benchmark captured in scripts in the repo.
uttampawar commentedon Jul 21, 2017
@rvagg @mhdawson Thank you. I'll start working on this script. It would be useful to have a mirror job for me to experiment once I've the scripts ready.
mhdawson commentedon Sep 15, 2017
Updated the main benchmarking jobs to include DC-EIS and configured to publish data to database for the charts @uttampawar had added.
@uttampawar I notice that you did not add charts for the pre/post footprint ? Was this on purpose ?
mhdawson commentedon Nov 2, 2017
I think this is complete, new issues can be opened for any follow on work, closing.